Controller, endoscope system, control method, and control program

ABSTRACT

A controller controls an image that is captured by an image sensor of an endoscope and is displayed on the display screen of a display device. The controller includes a processor. The processor acquires a first image that is an image captured by the image sensor, detects a first angle of a surgical instrument, and generates a second image rotated with respect to the first image on the basis of the first angle and a predetermined target angle. The first angle is an angle formed by the surgical instrument in the first image with respect to a predetermined reference line that is set with respect to a plane of the first image. A second angle formed by the surgical instrument in the second image with respect to the predetermined reference line is equal to the predetermined target angle.

PROGRAM Technical Field

The present invention relates to a controller, an endoscope system, acontrol method, and a control program, and particularly relates to acontroller, an endoscope system, a control method, and a controlprogram, by which an endoscope image displayed on a display device iscontrolled.

The present application claims priority under the provisional U.S. Pat.application No. 63/076408 filed on Sep. 10, 2020, which is incorporatedherein by reference. This is a continuation of International ApplicationPCT/JP2021/033209 which is hereby incorporated by reference herein inits entirety.

Background Art

Conventionally, a system has been proposed to move the field of view ofan endoscope in a semiautonomous manner by causing the endoscope tofollow a surgical instrument (for example, see PTL 1 and PTL 2).

Citation List Patent Literatures

-   [PTL 1] Japanese Unexamined Patent Application Publication No.    2003-127076-   [PTL 2] Japanese Unexamined Patent Application Publication No.    2001-112704

SUMMARY OF INVENTION

An aspect of the present invention is a controller configured to controlan image that is captured by an image sensor of an endoscope and isdisplayed on the display screen of a display device, the controllerincluding a processor, wherein the processor acquires a first image thatis an image captured by the image sensor, the processor detects a firstangle of a surgical instrument, the first angle being an angle formed bythe surgical instrument in the first image with respect to apredetermined reference line that is set with respect to a plane of thefirst image, and the processor generates a second image rotated withrespect to the first image on the basis of the first angle and apredetermined target angle, the surgical instrument forming a secondangle in the second image with respect to the predetermined referenceline such that the second angle is equal to the predetermined targetangle.

Another aspect of the present invention is an endoscope system includingan endoscope, a moving device that includes a robot arm and that movesthe endoscope in a subject, and the controller.

Another aspect of the present invention is a control method forcontrolling an image that is captured by an image sensor of an endoscopeand is displayed on the display screen of a display device, the controlmethod including: acquiring a first image that is an image captured bythe image sensor; detecting a first angle of a surgical instrument, thefirst angle being an angle formed by the surgical instrument in thefirst image with respect to a predetermined reference line that is setwith respect to a plane of the first image, and generating a secondimage rotated with respect to the first image on the basis of the firstangle and a predetermined target angle, the surgical instrument forminga second angle in the second image with respect to the predeterminedreference line such that the second angle is equal to the predeterminedtarget angle.

Another aspect of the present invention is a non-transitorycomputer-readable medium having a control program stored therein, theprogram being for controlling an image that is captured by an imagesensor of an endoscope and is displayed on a display screen of a displaydevice, the program causing a processor to execute functions of:acquiring a first image that is an image captured by the image sensor;detecting a first angle of a surgical instrument, the first angle beingan angle formed by the surgical instrument in the first image withrespect to a predetermined reference line that is set with respect to aplane of the first image; and generating a second image rotated withrespect to the first image on a basis of the first angle and apredetermined target angle, the surgical instrument forming a secondangle in the second image with respect to the predetermined referenceline such that the second angle is equal to the predetermined targetangle.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is an overall schematic diagram illustrating an example of anendoscope system according to a first embodiment of the presentinvention.

FIG. 1B is an overall schematic diagram illustrating another example ofthe endoscope system according to the first embodiment of the presentinvention.

FIG. 1C is an overall schematic diagram illustrating another example ofthe endoscope system according to the first embodiment of the presentinvention.

FIG. 2 is a block diagram of the endoscope system illustrated in FIGS.1A to 1C.

[FIG. 3A] FIG. 3A illustrates an example of an endoscope image displayedon the display screen of a display device.

FIG. 3B illustrates another example of an endoscope image displayed onthe display screen of a display device.

FIG. 3C illustrates another example of an endoscope image displayed onthe display screen of a display device.

FIG. 4 is a flowchart of a control method according to the firstembodiment of the present invention.

FIG. 5A illustrates an example of a first image.

FIG. 5B illustrates a second image rotated with respect to the firstimage in FIG. 5A.

FIG. 6A illustrates another example of the first image.

FIG. 6B illustrates a second image rotated with respect to the firstimage in FIG. 6A.

FIG. 7A illustrates an example of a third image acquired by a controllerin an endoscope system according to a second embodiment of the presentinvention.

FIG. 7B illustrates an example of a first image acquired by thecontroller in the endoscope system according to the second embodiment ofthe present invention.

FIG. 7C illustrates a second image rotated with respect to the firstimage in FIG. 7B.

FIG. 8 is a flowchart of a control method according to the secondembodiment of the present invention.

FIG. 9A illustrates an example of a first image acquired by a controllerin an endoscope system according to a third embodiment of the presentinvention.

FIG. 9B illustrates a second image rotated with respect to the firstimage in FIG. 9A.

FIG. 10 is a flowchart of a control method according to the thirdembodiment of the present invention.

FIG. 11A illustrates an example of a first image acquired by acontroller in an endoscope system according to a fourth embodiment ofthe present invention.

FIG. 11B illustrates another example of a first image acquired by thecontroller in the endoscope system according to the fourth embodiment ofthe present invention.

FIG. 11C illustrates a second image rotated with respect to the firstimage in FIG. 11B.

FIG. 12 is a flowchart of a control method according to the fourthembodiment of the present invention.

FIG. 13A illustrates an example of the first image acquired by thecontroller in a second rotation mode.

FIG. 13B illustrates another example of the first image acquired by thecontroller in the second rotation mode.

FIG. 13C illustrates a fourth image rotated with respect to the firstimage in FIG. 13B.

DESCRIPTION OF EMBODIMENTS First Embodiment

A controller, an endoscope system, a control method, and a controlprogram according to a first embodiment of the present invention will bedescribed below with reference to the accompanying drawings.

As illustrated in FIGS. 1A to 1C, an endoscope system 10 according tothe present embodiment is used for a surgical operation in which anendoscope 2 and at least one surgical instrument 6 are inserted into thebody of a patient P serving as a subject and an affected part is treatedwith the surgical instrument 6 while the surgical instrument 6 isobserved through the endoscope 2. The endoscope system 10 is used for,for example, laparoscopic surgery.

In the endoscope system 10 of FIG. 1A, the surgical instrument 6 is heldwith a hand of a surgeon and is manually operated by the surgeon. Theendoscope system 10 of FIGS. 1B and 1C includes a moving device 31 thatholds and moves the surgical instrument 6 and a controller 101 thatcontrols the moving device 31. As illustrated in FIG. 1B, the surgicalinstrument 6 may be connected to the tip of the robot arm of the movingdevice 31 and integrated with the robot arm. Alternatively, the surgicalinstrument 6 may be a separate part held by the robot arm.

As illustrated in FIG. 1A to 2, the endoscope system 10 includes theendoscope 2, a moving device 3 that holds the endoscope 2 and moves theendoscope 2 in the subject, an endoscope processor 4 that is connectedto the endoscope 2 and processes an endoscope image A captured by theendoscope 2, a controller 1 that is connected to the moving device 3 andthe endoscope processor 4 and controls the moving device 3, and adisplay device 5 that is connected to the endoscope processor 4 anddisplays the endoscope image A.

The endoscope 2 is, for example, a rigid endoscope and includes an imagesensor 2 a. The image sensor 2 a is, for example, a three-dimensionalcamera provided at the tip portion of the endoscope 2 and captures astereo image, which includes a tip 6 a of the surgical instrument 6, asthe endoscope image A (for example, see FIGS. 3A and 3B). The imagesensor 2 a is an image sensor such as a CCD (Charge Coupled Device)image sensor or a CMOS (Complementary Metal Oxide Semiconductor) imagesensor. The image sensor 2 a generates an image of a predeterminedregion by converting received light from the predetermined region intoan electric signal through photoelectric conversion. A stereo image asan endoscope image E is generated by performing image processing on twoimages with a parallax through the endoscope processor 4 or the like.The surgical instrument 6 has a long shaft 6 b. The surgical instrument6 may further include an end effector 6 c connected to the tip of theshaft 6 b.

The endoscope image A is transmitted from the endoscope 2 to theendoscope processor 4, is subjected to necessary processing in theendoscope processor 4, is transmitted from the endoscope processor 4 tothe display device 5, and is displayed on a display screen 5 a of thedisplay device 5. The display device 5 may be any display, for example,a liquid crystal display or an organic electroluminescent display. Asurgeon operates the surgical instrument 6 in a body while observing theendoscope image A displayed on the display screen 5 a. The displaydevice 5 may include an audio system, for example, a speaker.

In addition to the display device 5, a user terminal for communicationswith the controller 1 and the endoscope processor 4 via a communicationnetwork may be provided to display the endoscope image E at theterminal. The terminal is, for example, a notebook computer, a laptopcomputer, a tablet computer, or a smartphone but is not particularlylimited thereto.

The moving device 3 includes a robot arm 3 a (including an electricscope holder) that holds the endoscope 2 and three-dimensionallycontrols the position and orientation of the endoscope 2. The movingdevice 3 in FIGS. 1A to 1C includes the robot arm 3 a having a pluralityof joints 3 b that operate to move the endoscope 2, therebythree-dimensionally changing the position and orientation of theendoscope 2. The moving device 3 may further include a mechanism forrotating the endoscope 2 about a visual axis (optical axis).

As illustrated in FIG. 2 , the controller 1 includes at least oneprocessor 1 a like a central processing unit, a memory 1 b, a storageunit 1 c, an input interface 1 d, an output interface 1 e, and a userinterface 1 f. The controller 1 may be, for example, a desktop computer,a tablet computer, a laptop computer, a smartphone, or a cellular phone.

The storage unit 1 c is a hard disk or a nonvolatile recording mediumincluding a semiconductor memory such as flash memory and stores variousprograms including a follow-up control program (not illustrated) and animage control program (control program) 1 g and data necessary for theprocessing of the processor 1 a. Processing performed by the processor 1a may be partially implemented by dedicated logic circuits or hardware,for example, an FPGA (Field Programmable Gate Array), a SoC(System-on-a-Chip), an ASIC (Application Specific Integrated Circuit),and a PLD (Programmable Logic Device). The processing will be describedlater.

The storage unit 1 c may be a server, e.g., a cloud server connected viaa communication network to the controller 1 provided with acommunication interface, instead of a recording medium integrated in thecontroller 1. The communication network may be, for example, a publicnetwork such as the Internet, a dedicated line, or a LAN (Local AreaNetwork). The connection of the devices may be wired connection orwireless connection.

Any one of the configurations of the at least one processor 1 a, thememory 1 b, the storage unit 1 c, the input interface 1 d, the outputinterface 1 e, and the user interface 1 f in the controller 1 may beprovided for a user terminal, aside from the endoscope processor 4 andthe controller 1. The controller 1 may be integrated with the movingdevice 3.

The processor 1 a performs processing according to the follow-up controlprogram (not illustrated) read in the memory 1 b to cause the endoscope2 to follow the surgical instrument 6 to be followed. Specifically, theprocessor 1 a acquires the three-dimensional position of the tip 6 a ofthe surgical instrument 6 from the endoscope image A and controls themoving device 3 on the basis of the three-dimensional position of thetip 6 a and the three-dimensional position of a predetermined targetpoint set in the field of view of the endoscope 2. The target point is,for example, a point that is located on an optical axis and is disposedat a predetermined distance from the tip of the endoscope 2 in adirection parallel to the optical axis. The target point corresponds toa center point C of the endoscope image A. Thus, the controller 1controls a movement of the endoscope 2 and causes the endoscope 2 toautomatically follow the surgical instrument 6 such that the targetpoint is disposed at the tip 6 a.

The processor 1 a is configured to operate in a rotation mode. Asillustrated in FIGS. 3A to 3C, by performing processing in the rotationmode according to the image control program 1 g read in the memory 1 b,the processor 1 a controls the rotation angle of the endoscope image Adisplayed on the display screen 5 a and adjusts the vertical directionof the endoscope image A such that an angle θ of the surgical instrument6 is equal to a target angle θt on the display screen 5 a. FIGS. 3A to3C illustrate examples that are different from one another in theposition of the surgical instrument 6 and are identical to each other inthe angle θ of the surgical instrument 6 and the vertical direction ofthe endoscope image A. In FIGS. 3A to 3C, reference character Lh denotesa horizontal line extending in the lateral direction (horizontaldirection) of the display screen 5 a, and reference character Lv denotesa vertical line extending in the longitudinal direction (verticaldirection) of the display screen 5 a.

The input interface 1 d and the output interface 1 e are connected tothe endoscope processor 4. The controller 1 can acquire the endoscopeimage A from the endoscope 2 via the endoscope processor 4 and outputthe endoscope image A to the display device 5 via the endoscopeprocessor 4. The input interface 1 d may be directly connected to theendoscope 2 and the output interface 1 e may be directly connected tothe display device 5 such that the controller 1 can directly acquire theendoscope image A from the endoscope 2 and directly output the endoscopeimage A to the display device 5.

The user interface 1 f has input devices for inputs to the userinterface 1 f by users such as a surgeon and receives a user input. Theinput devices include a mouse, a button, a keyboard, and a touch panel.Moreover, the user interface 1 f has a means that allows a user toswitch on/off the rotation mode. The means is, for example, a switch. Atthe start of the controller 1, the switch is initially set to be turnedoff. Thereafter, when the switch is turned on by a user, the userinterface 1 f receives the input of the turn-on of the rotation mode.When the switch is turned off by the user, the user interface 1 freceives the input of the turn-off of the rotation mode.

For a proper operation of the surgical instrument 6 by the surgeon whois observing the endoscope image A, it is important to properly set thevertical direction of the endoscope image A (the orientations ofsubjects, for example, the surgical instrument 6 and a biological tissuein the endoscope image A) displayed on the display screen 5 a. Forexample, it is desirable to the surgeon that the surgical instrument 6operated with the right hand of the surgeon would protrude at about 30°from the right side of the endoscope image A. During a surgicaloperation, however, a movement of the surgical instrument 6 by thesurgeon or a change of the orientation of the endoscope 2 following thesurgical instrument 6 leads to a change of the angle θ of the surgicalinstrument 6 in the endoscope image A. When the surgical instrument 6 inthe endoscope image A is to be displayed at the target angle θt on thedisplay screen 5 a, a user, e.g., a surgeon can start the rotation modeby turning on the switch.

A control method performed by the processor 1 a in the rotation modewill be described below.

As indicated in FIG. 4 , the control method according to the presentembodiment includes step S1 of receiving the input of the turn-on of therotation mode, step S2 of acquiring a first image A1, step S3 ofdetecting a first angle θ1 of the surgical instrument 6 in the firstimage A1, steps S4 to S6 of generating a second image A2 that is rotatedwith respect to the first image A1 on the basis of the first angle θ1and the predetermined target angle θt, step S7 of displaying the secondimage A2 on the display screen 5 a, and step S8 of receiving the inputof the turn-off of the rotation mode.

When the user interface 1 f receives the input of the turn-on of therotation mode (YES at step S1), the processor 1 a acquires the firstimage A1, which is the latest endoscope image A, from the endoscope 2 asillustrated in FIGS. 5A and 6A (step S2). The top, bottom, left, andright of the first image A1 correspond to the top, bottom, left, andright of the display screen 5 a.

The processor 1 a then recognizes the surgical instrument 6 and theshaft 6 b in the first image A1 and detects the first angle θ1 of thesurgical instrument 6 (step S3). For recognizing the surgical instrument6 and the shaft 6 b, for example, a known image recognition techniqueaccording to deep learning is used. The first angle θ1 is an angleformed by a longitudinal axis B of the shaft 6 b with respect to apredetermined reference line L.

The predetermined reference line L is a straight line that is set withrespect to the plane of the first image A1 and forms a predeterminedangle with respect to the horizontal line in the first image A1. Thereference line L is fixed relative to the first image A1. In the presentembodiment, the reference line L is a horizontal line passing throughthe center point C of the first image A1 and corresponds to a horizontalline Lh of the display screen 5 a.

The processor 1 a then compares the first angle θ1 with thepredetermined target angle θt (step S4). The target angle θt is, forexample, a value determined in advance by the surgeon according to thepreferences of the surgeon, a fixed value set for each case of surgery(e.g., a surgical site), or a fixed value set for each surgicalinstrument 6. The target angle θt is stored in advance in, for example,the storage unit 1 c.

If the first angle θ1 is equal to the target angle θt (YES at step S4),the processor 1 a returns to step S2 without performing steps S5 to S7.In this case, the processor 1 a outputs the first image A1 to thedisplay device 5 and displays the image on the display screen 5 a.

If the first angle θ1 is different from the target angle θt (NO at stepS4), the processor 1 a then calculates a difference Δθ between the firstangle θ1 and the target angle θt (step S5).

Subsequently, as illustrated in FIGS. 5B and 6B, the processor 1 agenerates the second image A2 that is the endoscope image A rotated bythe difference Δθ with respect to the first image A1 (step S6). Thesecond image A2 is an image rotated about a predetermined rotation axisD with respect to the first image A1 by the difference Δθ in a directionthat eliminates the difference Δθ (clockwise in FIGS. 5B and 6B). Thus,a second angle θ2 formed by the longitudinal axis B of the shaft 6 b inthe second image A2 with respect to the reference line L is equal to thetarget angle θt. The rotation axis D is an axis that passes through thecenter point C of the first image A1 and is parallel to the optical axisof the endoscope 2 (that is, perpendicular to the plane of the firstimage A1).

FIGS. 5A and 5B illustrate the first image A1 and the second image A2,respectively, when the tip 6 a is disposed at the center point C. FIGS.6A and 6B illustrate the first image A1 and the second image A2,respectively, when the tip 6 a is disposed at a position deviated fromthe center point C.

In an example of step S6, the processor 1 a generates the second imageA2 by rotating the image sensor 2 a about the optical axis by thedifference Δθ. In other words, the image sensor 2 a captures the secondimage A2 that is the endoscope image A rotated by the difference Δθ withrespect to the first image A1.

In another example of step S6, the processor 1 a generates the secondimage A2 by rotating the first image A1 by the difference Δθ throughimage processing.

In another example of step S6, the processor 1 a generates the secondimage A2 by rotating the endoscope 2 by using a rotating mechanismprovided at the tip of the moving device 3.

The processor 1 a then outputs the generated second image A2 to thedisplay device 5 and displays the image on the display screen 5 a (stepS7). The surgical instrument 6 in the second image A2 displayed on thedisplay screen 5 a forms the target angle θt with respect to thehorizontal line Lh of the display screen 5 a.

Until the user interface 1 f receives the input of the turn-off of therotation mode in step S8, the processor 1 a regularly performs theacquisition of the first image A1 in step S2 and performs steps S3 to S7each time an additional first image A is acquired. Thus, steps S2 to S7are repeated and the angle θ of the surgical instrument 6 on the displayscreen 5 a is kept at the target angle θt while the rotation mode isexecuted.

As described above, according to the present embodiment, when the firstangle θ1 of the surgical instrument 6 in the first image A1, which isthe endoscope image A captured by the endoscope 2, is equal to thepredetermined target angle θt, the first image A1 is displayed on thedisplay screen 5 a. When the first angle θ1 is different from thepredetermined target angle θt, the second image A2 is automaticallygenerated by a rotation with respect to the first image A1 such that thesecond angle θ2 of the surgical instrument 6 is equal to the targetangle θt, and the second image A2 is displayed on the display screen 5 ainstead of the first image A1.

As described above, the vertical directions of the endoscope images A1and A2 to be displayed on the display screen 5 a are automaticallycontrolled by the processor 1 a such that the angle θ of the surgicalinstrument 6 on the display screen 5 a is equal to the target angle θt.Thus, the surgeon who is operating the surgical instrument 6 does notneed to manually operate the endoscope 2 to adjust the verticaldirection of the endoscope image A. In other words, the endoscope imagesA1 and A2 can be provided in proper vertical directions so as tofacilitate procedures for the surgeon without the need for taking asurgeon’s hand off from the surgical instrument 6.

Second Embodiment

A controller, an endoscope system, a control method, and a controlprogram according to a second embodiment of the present invention willbe described below with reference to the accompanying drawings.

As illustrated in FIGS. 7A and 7B, the present embodiment is differentfrom the first embodiment in that a user sets a target angle θt and arotation angle Δθ of a second image A2 with respect to a first image A1by using the angle of a surgical instrument 6 in an endoscope image A.In the present embodiment, configurations different from those of thefirst embodiment will be described. Configurations in common with thefirst embodiment are indicated by the same reference numerals and anexplanation thereof is omitted.

As in the first embodiment, an endoscope system 10 according to thepresent embodiment includes a controller 1, an endoscope 2, a movingdevice 3, an endoscope processor 4, and a display device 5.

A user interface 1 f is configured to receive the inputs of a firsttrigger and a second trigger from a user. For example, the userinterface 1 f includes a first switch for a user input of the firsttrigger and a second switch for a user input of the second trigger.

In a rotation mode of the present embodiment, a processor 1 a performs acontrol method shown in FIG. 8 .

The control method according to the present embodiment includes step S1,step S11 of receiving the first trigger, step S12 of acquiring a thirdimage A3, step S13 of detecting a third angle θ3 of the surgicalinstrument 6 in the third image A3, step S14 of setting the target angleθt at the third angle θ3, step S15 of receiving the second trigger, andsteps S2, S3, and S5 to S8.

When the user interface 1 f receives the input of the turn-on of therotation mode (YES at step S1), the processor 1 a waits for thereception of the first trigger by the user interface 1 f (step S11). Asillustrated in FIG. 7A, a surgeon moves the surgical instrument 6 so asto place the surgical instrument 6 at a desired angle θ3 in theendoscope image A displayed on a display screen 5 a and inputs the firsttrigger to the user interface 1 f.

In response to the reception of the first trigger by the user interface1 f (YES at step S11), the processor 1 a acquires the third image A3,which is the latest endoscope image A, from the endoscope 2 (step S12).

The processor 1 a then detects the third angle θ3 of the surgicalinstrument 6 in the third image A3 according to the same method as stepS3 (step S13). Like a first angle θ1, the third angle θ3 is an angleformed by a longitudinal axis B of a shaft 6 b with respect to apredetermined reference line L.

The processor 1 a then sets the target angle θt at the third angle θ3(step S14).

Thereafter, the processor 1 a waits for the reception of the secondtrigger by the user interface 1 f (step S15). As illustrated in FIG. 7B,the surgeon moves the surgical instrument 6 so as to change the angle ofthe surgical instrument 6 by a desired angle Δθ from the third angle θ3in the endoscope image A displayed on the display screen 5 a and inputsthe second trigger to the user interface 1 f.

In response to the reception of the second trigger by the user interface1 f (YES at step S15), the processor 1 a acquires the first image, whichis the latest endoscope image A (step S2), and detects the first angleθ1 of the surgical instrument 6 in the first image A1 (step S3).

Subsequently, the processor 1 a calculates a difference Δθ between thefirst angle θ1 and the target angle θt (step S5) and generates, asillustrated in FIG. 7C, the second image A2 that is the endoscope imageA rotated by the difference Δθ with respect to the first image A1 (stepS6). The second image A2 is the endoscope image A displayed against abackground including a biological tissue E present behind the surgicalinstrument 6, the background being rotated by an angle Δθ desired by auser with respect to the third image A3. A second angle θ2 of thesurgical instrument 6 in the second image A2 is equal to the targetangle θt that is the third angle θ3 of the surgical instrument 6 in thethird image A3.

Thereafter, as in the first embodiment, steps S2 to S7 are repeated andthe angle θ of the surgical instrument 6 on the display screen 5 a iskept at the target angle θt.

As described above, according to the present embodiment, the surgeon canset the target angle θt and the difference Δθ at desired values by usingthe surgical instrument 6 in the endoscope image A. Moreover, thesurgeon can rotate the background of the surgical instrument 6 by adesired angle A6 and properly adjust the orientation of the background.For example, when the biological tissue E is diagonally laid in theendoscope image A as illustrated in FIG. 7A, the vertical direction ofthe endoscope image A displayed on the display screen 5 a can beadjusted so as to place the biological tissue E in the horizontaldirection suitable for a procedure as illustrated in FIG. 7C. Thus, therotation mode of the present embodiment is effective for adjusting theangle of the background in the endoscope image A displayed on thedisplay screen 5 a.

Furthermore, after the target angle θt and the difference Δθ are set,the vertical directions of the endoscope images A1 and A2 to bedisplayed on the display screen 5 a are automatically controlled by theprocessor 1 a as in the first embodiment such that the angle θ of thesurgical instrument 6 on the display screen 5 a is equal to the targetangle θt. Hence, the endoscope images A1 and A2 can be provided inproper vertical directions so as to facilitate procedures for thesurgeon without the need for taking a surgeon’s hand off from thesurgical instrument 6.

Third Embodiment

A controller, an endoscope system, a control method, and a controlprogram according to a third embodiment of the present invention will bedescribed below with reference to the accompanying drawings.

As illustrated in FIGS. 9A and 9B, the present embodiment is differentfrom the first embodiment in that an angle θ of a surgical instrument 6is the deflection angle of a tip 6 a. In the present embodiment,configurations different from those of the first embodiment will bedescribed. Configurations in common with the first embodiment areindicated by the same reference numerals and an explanation thereof isomitted.

As in the first embodiment, an endoscope system 10 according to thepresent embodiment includes a controller 1, an endoscope 2, a movingdevice 3, an endoscope processor 4, and a display device 5.

In a rotation mode of the present embodiment, a processor 1 a controlsthe rotation angle of an endoscope image A displayed on a display screen5 a such that the tip 6 a of the surgical instrument 6 is disposed on apredetermined horizontal line Lh passing through the center of thedisplay screen 5 a. Specifically, the processor 1 a performs a controlmethod shown in FIG. 10 .

The control method according to the present embodiment includes steps S1and S2, step S21 of determining whether the tip 6 a of the surgicalinstrument 6 is disposed on a predetermined reference line L, and stepsS3 and S6 to S8.

As in the first embodiment, in response to the reception of the input ofthe turn-on of the rotation mode by a user interface 1 f (YES at stepS1), the processor 1 a acquires a first image A1 (step S2).

The processor 1 a then recognizes the surgical instrument 6 in the firstimage A1 and determines whether the tip 6 a of the surgical instrument 6is disposed on the predetermined reference line L (step S21). Thepredetermined reference line L is a horizontal line passing through acenter point C of the first image A1 and corresponds to a predeterminedhorizontal line Lh passing through the center point of the displayscreen 5 a.

If the tip 6 a is disposed on the reference line L (YES at step S21),the processor 1 a returns to step S2 without performing steps S3 and S6to S8. In this case, the processor 1 a outputs the first image A1 to thedisplay device 5 and displays the image on the display screen 5 a.

As illustrated in FIG. 9A, when the tip 6 a is not disposed on thereference line L (NO at step S21), the processor 1 a then detects afirst angle θ1 of the surgical instrument 6 in the first image A1. Thefirst angle θ1 is an angle formed by a line F connecting the centerpoint C and the tip 6 a with respect to the reference line L.

Subsequently, the processor 1 a generates a second image A2 that is theendoscope image A rotated by the first angle θ1 with respect to thefirst image A1 (step S6). In other words, in the present embodiment, atarget angle θt is 0°. In the second image A2, the tip 6 a has adeflection angle (second angle) of 0° and is disposed on thepredetermined reference line L.

Thereafter, steps S2, S21, S3, S6, and S7 are repeated, so that theangle θ of the surgical instrument 6 on the display screen 5 a is keptat 0° and the tip 6 a of the surgical instrument 6 is kept on thehorizontal line Lh.

As described above, according to the present embodiment, when the tip 6a of the surgical instrument 6 in the first image A1, which is theendoscope image A captured by the endoscope 2, is disposed on thereference line L, the first image A1 is displayed on the display screen5 a. When the tip 6 a is not disposed on the reference line L, thesecond image A2 rotated with respect to the first image A1 isautomatically generated with the tip 6 a disposed on the reference lineL. The second image A2 is displayed on the display screen 5 a instead ofthe first image A1.

As described above, the vertical directions of the endoscope images A1and A2 to be displayed on the display screen 5 a are automaticallycontrolled by the processor 1 a such that the tip 6 a of the surgicalinstrument 6 on the display screen 5 a is disposed on the predeterminedhorizontal line Lh. Thus, the surgeon who is operating the surgicalinstrument 6 does not need to manually operate the endoscope 2 to adjustthe vertical direction of the endoscope image A. In other words, theendoscope images A1 and A2 can be provided in proper vertical directionsso as to facilitate procedures for the surgeon without the need fortaking a surgeon’s hand off from the surgical instrument 6.

In the present embodiment, the predetermined reference line L is ahorizontal line passing through the center point C. The direction andposition of the predetermined reference line L can be optionallychanged.

For example, if the tip 6 a is to be disposed on a vertical line Lvpassing through the center point of the display screen 5 a, thepredetermined reference line L may be a vertical line passing throughthe center point C. Alternatively, if the tip 6 a is to be disposed onan inclined straight line passing through the center point of thedisplay screen 5 a, the predetermined reference line L may be aninclined straight line passing through the center point C.

Fourth Embodiment

A controller, an endoscope system, a control method, and a controlprogram according to a fourth embodiment of the present invention willbe described below with reference to the accompanying drawings.

As illustrated in FIGS. 11A to 11C, the present embodiment is differentfrom the first embodiment in that a second image A2 is generated when adeviation of a first angle θ1 of a surgical instrument 6 in a firstimage A1 from a target angle θt exceeds a predetermined threshold valueX. In the present embodiment, configurations different from those of thefirst embodiment will be described. Configurations in common with thefirst embodiment are indicated by the same reference numerals and anexplanation thereof is omitted.

As in the first embodiment, an endoscope system 10 according to thepresent embodiment includes a controller 1, an endoscope 2, a movingdevice 3, an endoscope processor 4, and a display device 5.

In the present embodiment, while the controller 1 controls the movingdevice 3, a processor 1 a always operates in a rotation mode andperforms a control method shown in FIG. 12 .

The control method according to the present embodiment includes steps S2and S3, step S31 of determining whether the absolute value of adifference between the first angle θ1 and the target angle θt is at mosta predetermined threshold value, and steps S5 to S7.

The processor 1 a acquires the first image A1, which is the latestendoscope image A, from the endoscope 2 (step S2) and detects the firstangle θ1 of the surgical instrument 6 in the first image A1 (step S3).

The processor 1 a then determines whether the first angle θ1 is within arange of the target angle θt±X (step S31).

As illustrated in FIG. 11A, if the first angle θ1 is within a range ofthe target angle θt±X (YES at step S31), that is, if the absolute valueof a difference between the angles θ1 and θt is equal to or smaller thanthe threshold value X, the processor 1 a returns to step S2 withoutperforming steps S5 to S7. In this case, the processor 1 a outputs thefirst image A1 to the display device 5 and displays the image on thedisplay screen 5 a.

As illustrated in FIG. 11B, if the first angle θ1 is out of a range ofthe target angle θt±X (NO at step S31), that is, if the absolute valueof a difference between the angles θ1 and θt is larger than thethreshold value X, the processor 1 a then calculates a difference Δθbetween the first angle θ1 and the target angle θt (step S5) andgenerates the second image A2 that is the endoscope image A rotated bythe difference Δθ with respect to the first image A1 (step S6).

As in the first embodiment, by repeating steps S2 to S7, the endoscopeimage A displayed on the display screen 5 a is rotated each time thefirst angle θ1 deviates from a range of the target angle θt±X, so thatthe angle of the surgical instrument 6 on the display screen 5 a is keptin a predetermined range including the target angle θt.

As described above, according to the present embodiment, when the firstangle θ1 of the surgical instrument 6 in the first image A1, which isthe endoscope image A captured by the endoscope 2, is within a range ofthe target angle θt±X, the first image A1 is displayed on the displayscreen 5 a. When the first angle θ1 deviates from a range of the targetangle θt±X, the second image A2 is automatically generated by a rotationwith respect to the first image A1 such that the second angle θ2 of thesurgical instrument 6 is equal to the target angle θt, and the secondimage A2 is displayed on the display screen 5 a instead of the firstimage A1.

As described above, the vertical directions of the endoscope images A1and A2 to be displayed on the display screen 5 a are automaticallycontrolled by the processor 1 a such that the angle of the surgicalinstrument 6 on the display screen 5 a is within a range of the targetangle θt±X. Thus, the surgeon who is operating the surgical instrument 6does not need to manually operate the endoscope 2 to adjust the verticaldirection of the endoscope image A. In other words, the endoscope imagesA1 and A2 can be provided in proper vertical directions so as tofacilitate procedures for the surgeon without the need for taking asurgeon’s hand off from the surgical instrument 6.

Also in the present embodiment, the control method may include steps S1and S8 as in the first to third embodiments. The processor 1 a may startand end the rotation mode in response to the inputs of the turn-on andturn-off of the rotation mode to a user interface 1 f.

In the foregoing embodiments, the predetermined reference line L is ahorizontal line of the first image A1 but is not limited thereto. A lineextending in any direction can be set as the reference line L.

For example, the predetermined reference line L may be a vertical lineextending in the longitudinal direction (vertical direction) of thefirst image A1 or a line extending in an oblique direction of the firstimage A1.

In the foregoing embodiments, the processor 1 a generates the secondimage A2 rotated about the rotation axis D that passes through thecenter point C of the first image A1 and is parallel to the opticalaxis. The direction and position of the rotation axis D can beoptionally changed depending upon a user request or the like. In otherwords, the rotation axis D may be inclined with respect to the opticalaxis and may pass through a position deviated from the center point C.

In the foregoing embodiments, the processor 1 a may be configured tooperate in any one of a first rotation mode in which the second image A2is generated on the basis of the first angle θ1 of the surgicalinstrument 6 in the first image A1 and a second rotation mode in which afourth image A4 is generated on the basis of anatomical characteristicsin the first image A1. The processor 1 a performs the second rotationmode in response to the reception of the input of the turn-off of therotation mode by the user interface 1 f.

In the first rotation mode, the processor 1 a performs any one of thecontrol methods according to the first to fourth embodiments.

As illustrated in FIGS. 13A and 13B, a movement of the endoscope 2 in asubject changes the angle of an anatomical characteristic G of abiological tissue in the endoscope image A displayed on the displayscreen 5 a. The angle of the anatomical characteristic G is a rotationangle around the center point C of the endoscope image A. The secondrotation mode is a mode in which the anatomical characteristic G in theendoscope image A is displayed at a predetermined target angle θs on thedisplay screen 5 a. The second rotation mode is used for a scene otherthan a procedure for an affected part, for example, before and after aprocedure is performed on an affected part by the surgical instrument 6.

In the storage unit 1 c, a predetermined target angle θs of theanatomical characteristic G in the endoscope image A is stored for eachtype of the anatomical characteristic G. In the second rotation mode,the processor 1 a acquires the first image A1 from the endoscope 2,detects the anatomical characteristic G of a biological tissue in thefirst image A1, and recognizes the type of the anatomical characteristicG. Thereafter, as illustrated in FIG. 13C, the processor 1 a generatesthe fourth image A4 that is rotated with respect to the first image A1and has the anatomical characteristic G with an angle equal to thetarget angle θs, on the basis of the angle of the anatomicalcharacteristic G in the first image A1 and the target angle θs of thetype of the anatomical characteristic G stored in the storage unit 1 c.For example, if the anatomical characteristic G is an aorta, the targetangle θs is an angle of an aorta horizontally placed in the lower partof the endoscope image A.

When the vertical direction of the endoscope image A displayed on thedisplay screen 5 a changes, the layout of organs and biological tissuesare viewed differently. Hence, in order to facilitate the recognition oforgans or biological tissues in the endoscope image A by a surgeon, itis important to properly set the vertical direction of the endoscopeimage A displayed on the display screen 5 a. When the first rotationmode is turned off, the processor 1 a controls the vertical direction ofthe endoscope image A displayed on the display screen 5 a in the secondrotation mode. This can properly adjust the vertical direction of theendoscope image A displayed on the display screen 5 a, therebydisplaying the anatomical characteristic G of the biological tissue atthe predetermined target angle θs on the display screen 5 a.

The first to fourth embodiments may be implemented in combination asappropriate. For example, the processor 1 a may be operable in the fourrotation modes described in the first to fourth embodiments. In thiscase, a user may input selected one of the four rotation modes to theuser interface 1 f, and the processor 1 a may perform the inputtedrotation mode.

REFERENCE SIGNS LIST

-   1 Controller-   1 a Processor-   1 c Storage unit-   If User interface-   1 g Control program-   2 Endoscope-   3 Moving device-   5 Display device-   5 a Display screen-   6 Surgical instrument-   6 b Shaft-   10 Endoscope system-   θ1 First angle-   θ2 Second angle-   θ3 Third angle-   θt Target angle-   Δθ Difference-   A Endoscope image (image)-   A1 Endoscope image, first image-   A2 Endoscope image, second image-   A3 Endoscope image, third image-   A4 Endoscope image, fourth image-   B Shaft longitudinal axis-   C Center point-   D Rotation axis-   G Anatomical characteristic-   L Reference line-   Lh Horizontal line

1. A controller configured to control an image that is captured by animage sensor of an endoscope and is displayed on a display screen of adisplay device, the controller comprising a processor, wherein theprocessor acquires a first image that is an image captured by the imagesensor, the processor detects a first angle of a surgical instrument,the first angle being an angle formed by the surgical instrument in thefirst image with respect to a predetermined reference line that is setwith respect to a plane of the first image, and the processor generatesa second image rotated with respect to the first image on a basis of thefirst angle and a predetermined target angle, the surgical instrumentforming a second angle in the second image with respect to thepredetermined reference line such that the second angle is equal to thepredetermined target angle.
 2. The controller according to claim 1,wherein the processor generates the second image by rotating the imagesensor.
 3. The controller according to claim 1, wherein the processorgenerates the second image by rotating the first image through imageprocessing.
 4. The controller according to claim 1, wherein the targetangle is 0°.
 5. The controller according to claim 1, wherein theprocessor rotates the first image about a rotation axis parallel to anoptical axis of the endoscope.
 6. The controller according to claim 5,wherein the rotation axis passes through a center point of the firstimage.
 7. The controller according to claim 1, wherein the first angleis an angle formed by a longitudinal axis of a shaft of the surgicalinstrument in the first image with respect to the reference line, andthe second angle is an angle formed by the longitudinal axis of theshaft of the surgical instrument in the second image with respect to thereference line.
 8. The controller according to claim 1, wherein thereference line is a straight line that forms a predetermined angle withrespect to a horizontal line in the first image and is fixed relative tothe first image.
 9. The controller according to claim 1, wherein thereference line is a horizontal line that is set in the first image andcorresponds to a horizontal line on the display screen.
 10. Thecontroller according to claim 1, wherein the reference line is avertical line that is set in the first image and corresponds to avertical line on the display screen.
 11. The controller according toclaim 1, wherein the processor generates the second image when anabsolute value of a difference between the first angle and the targetangle is larger than a threshold value.
 12. The controller according toclaim 1, further comprising a user interface that receives an input of atrigger from a user, wherein the processor acquires a third image inresponse to reception of a first trigger by the user interface, thethird image being an image captured by the image sensor, and theprocessor sets the target angle at a third angle of the surgicalinstrument in the third image, the third angle being an angle formed bythe surgical instrument in the third image with respect to the referenceline.
 13. The controller according to claim 12, wherein after the targetangle is set, the processor acquires the first image in response toreception of a second trigger by the user interface.
 14. The controlleraccording to claim 1, wherein the processor sets an angle of thesurgical instrument at the target angle, the surgical instrument havinga tip disposed on a predetermined horizontal line on the display screen.15. The controller according to claim 1, wherein the processor isoperable in a rotation mode, and in the rotation mode, the processorregularly acquires the first image, detects the first angle, andgenerates the second image.
 16. The controller according to claim 15,further comprising a user interface that receives an input ofturn-on/turn-off of the rotation mode.
 17. The controller according toclaim 16, further comprising a storage unit that stores thepredetermined target angle of an anatomical characteristic for each typeof the anatomical characteristic, wherein when the user interfacereceives the input of the turn-off of the rotation mode, the processorrecognizes the type of the anatomical characteristic in the first image,and the processor generates a fourth image rotated with respect to thefirst image on a basis of the predetermined target angle of therecognized type, the predetermined target angle being stored in thestorage unit.
 18. An endoscope system comprising: an endoscope; a movingdevice that comprises a robot arm and that moves the endoscope in asubject; and the controller according to claim
 1. 19. A control methodfor controlling an image that is captured by an image sensor of anendoscope and is displayed on a display screen of a display device, thecontrol method comprising: acquiring a first image that is an imagecaptured by the image sensor; detecting a first angle of a surgicalinstrument, the first angle being an angle formed by the surgicalinstrument in the first image with respect to a predetermined referenceline that is set with respect to a plane of the first image; andgenerating a second image rotated with respect to the first image on abasis of the first angle and a predetermined target angle, the surgicalinstrument forming a second angle in the second image with respect tothe predetermined reference line such that the second angle is equal tothe predetermined target angle.
 20. A non-transitory computer-readablemedium having a control program stored therein, the program being forcontrolling an image that is captured by an image sensor of an endoscopeand is displayed on a display screen of a display device, the programcausing a processor to execute functions of: acquiring a first imagethat is an image captured by the image sensor; detecting a first angleof a surgical instrument, the first angle being an angle formed by thesurgical instrument in the first image with respect to a predeterminedreference line that is set with respect to a plane of the first image;and generating a second image rotated with respect to the first image ona basis of the first angle and a predetermined target angle, thesurgical instrument forming a second angle in the second image withrespect to the predetermined reference line such that the second angleis equal to the predetermined target angle.